{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dropout\n",
    "Dropout [1] is a technique for regularizing neural networks by randomly setting some features to zero during the forward pass. In this exercise you will implement a dropout layer and modify your fully-connected network to optionally use dropout.\n",
    "\n",
    "[1] [Geoffrey E. Hinton et al, \"Improving neural networks by preventing co-adaptation of feature detectors\", arXiv 2012](https://arxiv.org/abs/1207.0580)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "run the following from the cs231n directory and try again:\n",
      "python setup.py build_ext --inplace\n",
      "You may also need to restart your iPython kernel\n"
     ]
    }
   ],
   "source": [
    "# As usual, a bit of setup\n",
    "from __future__ import print_function\n",
    "import time\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from cs231n.classifiers.fc_net import *\n",
    "from cs231n.data_utils import get_CIFAR10_data\n",
    "from cs231n.gradient_check import eval_numerical_gradient, eval_numerical_gradient_array\n",
    "from cs231n.solver import Solver\n",
    "\n",
    "%matplotlib inline\n",
    "plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots\n",
    "plt.rcParams['image.interpolation'] = 'nearest'\n",
    "plt.rcParams['image.cmap'] = 'gray'\n",
    "\n",
    "# for auto-reloading external modules\n",
    "# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "def rel_error(x, y):\n",
    "  \"\"\" returns relative error \"\"\"\n",
    "  return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_train:  (49000, 3, 32, 32)\n",
      "y_train:  (49000,)\n",
      "X_val:  (1000, 3, 32, 32)\n",
      "y_val:  (1000,)\n",
      "X_test:  (1000, 3, 32, 32)\n",
      "y_test:  (1000,)\n"
     ]
    }
   ],
   "source": [
    "# Load the (preprocessed) CIFAR10 data.\n",
    "\n",
    "data = get_CIFAR10_data()\n",
    "for k, v in data.items():\n",
    "  print('%s: ' % k, v.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dropout forward pass\n",
    "In the file `cs231n/layers.py`, implement the forward pass for dropout. Since dropout behaves differently during training and testing, make sure to implement the operation for both modes.\n",
    "\n",
    "Once you have done so, run the cell below to test your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running tests with p =  0.25\n",
      "Mean of input:  10.000207878477502\n",
      "Mean of train-time output:  10.014059116977283\n",
      "Mean of test-time output:  10.000207878477502\n",
      "Fraction of train-time output set to zero:  0.749784\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n",
      "Running tests with p =  0.4\n",
      "Mean of input:  10.000207878477502\n",
      "Mean of train-time output:  9.977917658761159\n",
      "Mean of test-time output:  10.000207878477502\n",
      "Fraction of train-time output set to zero:  0.600796\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n",
      "Running tests with p =  0.7\n",
      "Mean of input:  10.000207878477502\n",
      "Mean of train-time output:  9.987811912159426\n",
      "Mean of test-time output:  10.000207878477502\n",
      "Fraction of train-time output set to zero:  0.30074\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "x = np.random.randn(500, 500) + 10\n",
    "\n",
    "for p in [0.25, 0.4, 0.7]:\n",
    "    out, _ = dropout_forward(x, {'mode': 'train', 'p': p})\n",
    "    out_test, _ = dropout_forward(x, {'mode': 'test', 'p': p})\n",
    "\n",
    "    print('Running tests with p = ', p)\n",
    "    print('Mean of input: ', x.mean())\n",
    "    print('Mean of train-time output: ', out.mean())\n",
    "    print('Mean of test-time output: ', out_test.mean())\n",
    "    print('Fraction of train-time output set to zero: ', (out == 0).mean())\n",
    "    print('Fraction of test-time output set to zero: ', (out_test == 0).mean())\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dropout backward pass\n",
    "In the file `cs231n/layers.py`, implement the backward pass for dropout. After doing so, run the following cell to numerically gradient-check your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dx relative error:  5.44560814873387e-11\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "x = np.random.randn(10, 10) + 10\n",
    "dout = np.random.randn(*x.shape)\n",
    "\n",
    "dropout_param = {'mode': 'train', 'p': 0.2, 'seed': 123}\n",
    "out, cache = dropout_forward(x, dropout_param)\n",
    "dx = dropout_backward(dout, cache)\n",
    "dx_num = eval_numerical_gradient_array(lambda xx: dropout_forward(xx, dropout_param)[0], x, dout)\n",
    "\n",
    "# Error should be around e-10 or less\n",
    "print('dx relative error: ', rel_error(dx, dx_num))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inline Question 1:\n",
    "What happens if we do not divide the values being passed through inverse dropout by `p` in the dropout layer? Why does that happen?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Answer:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fully-connected nets with Dropout\n",
    "In the file `cs231n/classifiers/fc_net.py`, modify your implementation to use dropout. Specifically, if the constructor of the net receives a value that is not 1 for the `dropout` parameter, then the net should add dropout immediately after every ReLU nonlinearity. After doing so, run the following to numerically gradient-check your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running check with dropout =  1\n",
      "Initial loss:  2.3004790897684924\n",
      "W0 relative error: 1.48e-07\n",
      "W1 relative error: 2.21e-05\n",
      "W2 relative error: 3.53e-07\n",
      "b0 relative error: 5.38e-09\n",
      "b1 relative error: 2.09e-09\n",
      "b2 relative error: 5.80e-11\n",
      "Running check with dropout =  0.75\n",
      "Initial loss:  2.302371489704412\n",
      "W0 relative error: 1.90e-07\n",
      "W1 relative error: 4.76e-06\n",
      "W2 relative error: 2.60e-08\n",
      "b0 relative error: 4.73e-09\n",
      "b1 relative error: 1.82e-09\n",
      "b2 relative error: 1.70e-10\n",
      "Running check with dropout =  0.5\n",
      "Initial loss:  2.3042759220785896\n",
      "W0 relative error: 3.11e-07\n",
      "W1 relative error: 1.84e-08\n",
      "W2 relative error: 5.35e-08\n",
      "b0 relative error: 2.58e-08\n",
      "b1 relative error: 2.99e-09\n",
      "b2 relative error: 1.13e-10\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "N, D, H1, H2, C = 2, 15, 20, 30, 10\n",
    "X = np.random.randn(N, D)\n",
    "y = np.random.randint(C, size=(N,))\n",
    "\n",
    "for dropout in [1, 0.75, 0.5]:\n",
    "    print('Running check with dropout = ', dropout)\n",
    "    model = FullyConnectedNet([H1, H2], input_dim=D, num_classes=C,\n",
    "                            weight_scale=5e-2, dtype=np.float64,\n",
    "                            dropout=dropout, seed=123)\n",
    "\n",
    "    loss, grads = model.loss(X, y)\n",
    "    print('Initial loss: ', loss)\n",
    "  \n",
    "  # Relative errors should be around e-6 or less; Note that it's fine\n",
    "  # if for dropout=1 you have W2 error be on the order of e-5.\n",
    "    for name in sorted(grads):\n",
    "        f = lambda _: model.loss(X, y)[0]\n",
    "        grad_num = eval_numerical_gradient(f, model.params[name], verbose=False, h=1e-5)\n",
    "        print('%s relative error: %.2e' % (name, rel_error(grad_num, grads[name])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regularization experiment\n",
    "As an experiment, we will train a pair of two-layer networks on 500 training examples: one will use no dropout, and one will use a keep probability of 0.25. We will then visualize the training and validation accuracies of the two networks over time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "(Iteration 1 / 125) loss: 7.856643\n",
      "(Epoch 0 / 25) train acc: 0.268000; val_acc: 0.186000\n",
      "(Epoch 1 / 25) train acc: 0.420000; val_acc: 0.255000\n",
      "(Epoch 2 / 25) train acc: 0.570000; val_acc: 0.300000\n",
      "(Epoch 3 / 25) train acc: 0.560000; val_acc: 0.245000\n",
      "(Epoch 4 / 25) train acc: 0.696000; val_acc: 0.277000\n",
      "(Epoch 5 / 25) train acc: 0.742000; val_acc: 0.297000\n",
      "(Epoch 6 / 25) train acc: 0.770000; val_acc: 0.271000\n",
      "(Epoch 7 / 25) train acc: 0.836000; val_acc: 0.250000\n",
      "(Epoch 8 / 25) train acc: 0.904000; val_acc: 0.259000\n",
      "(Epoch 9 / 25) train acc: 0.928000; val_acc: 0.279000\n",
      "(Epoch 10 / 25) train acc: 0.914000; val_acc: 0.246000\n",
      "(Epoch 11 / 25) train acc: 0.926000; val_acc: 0.281000\n",
      "(Epoch 12 / 25) train acc: 0.966000; val_acc: 0.288000\n",
      "(Epoch 13 / 25) train acc: 0.974000; val_acc: 0.290000\n",
      "(Epoch 14 / 25) train acc: 0.982000; val_acc: 0.298000\n",
      "(Epoch 15 / 25) train acc: 0.986000; val_acc: 0.298000\n",
      "(Epoch 16 / 25) train acc: 0.988000; val_acc: 0.294000\n",
      "(Epoch 17 / 25) train acc: 0.972000; val_acc: 0.294000\n",
      "(Epoch 18 / 25) train acc: 0.998000; val_acc: 0.310000\n",
      "(Epoch 19 / 25) train acc: 0.980000; val_acc: 0.307000\n",
      "(Epoch 20 / 25) train acc: 0.998000; val_acc: 0.302000\n",
      "(Iteration 101 / 125) loss: 0.001241\n",
      "(Epoch 21 / 25) train acc: 0.994000; val_acc: 0.304000\n",
      "(Epoch 22 / 25) train acc: 0.996000; val_acc: 0.306000\n",
      "(Epoch 23 / 25) train acc: 1.000000; val_acc: 0.304000\n",
      "(Epoch 24 / 25) train acc: 1.000000; val_acc: 0.305000\n",
      "(Epoch 25 / 25) train acc: 1.000000; val_acc: 0.307000\n",
      "0.25\n",
      "(Iteration 1 / 125) loss: 17.318479\n",
      "(Epoch 0 / 25) train acc: 0.220000; val_acc: 0.174000\n",
      "(Epoch 1 / 25) train acc: 0.376000; val_acc: 0.249000\n",
      "(Epoch 2 / 25) train acc: 0.416000; val_acc: 0.248000\n",
      "(Epoch 3 / 25) train acc: 0.498000; val_acc: 0.282000\n",
      "(Epoch 4 / 25) train acc: 0.562000; val_acc: 0.287000\n",
      "(Epoch 5 / 25) train acc: 0.606000; val_acc: 0.284000\n",
      "(Epoch 6 / 25) train acc: 0.626000; val_acc: 0.284000\n",
      "(Epoch 7 / 25) train acc: 0.644000; val_acc: 0.297000\n",
      "(Epoch 8 / 25) train acc: 0.706000; val_acc: 0.283000\n",
      "(Epoch 9 / 25) train acc: 0.736000; val_acc: 0.309000\n",
      "(Epoch 10 / 25) train acc: 0.764000; val_acc: 0.317000\n",
      "(Epoch 11 / 25) train acc: 0.784000; val_acc: 0.318000\n",
      "(Epoch 12 / 25) train acc: 0.816000; val_acc: 0.306000\n",
      "(Epoch 13 / 25) train acc: 0.844000; val_acc: 0.312000\n",
      "(Epoch 14 / 25) train acc: 0.856000; val_acc: 0.330000\n",
      "(Epoch 15 / 25) train acc: 0.856000; val_acc: 0.326000\n",
      "(Epoch 16 / 25) train acc: 0.882000; val_acc: 0.334000\n",
      "(Epoch 17 / 25) train acc: 0.890000; val_acc: 0.326000\n",
      "(Epoch 18 / 25) train acc: 0.900000; val_acc: 0.338000\n",
      "(Epoch 19 / 25) train acc: 0.896000; val_acc: 0.334000\n",
      "(Epoch 20 / 25) train acc: 0.902000; val_acc: 0.331000\n",
      "(Iteration 101 / 125) loss: 3.256056\n",
      "(Epoch 21 / 25) train acc: 0.920000; val_acc: 0.322000\n",
      "(Epoch 22 / 25) train acc: 0.924000; val_acc: 0.317000\n",
      "(Epoch 23 / 25) train acc: 0.932000; val_acc: 0.326000\n",
      "(Epoch 24 / 25) train acc: 0.938000; val_acc: 0.329000\n",
      "(Epoch 25 / 25) train acc: 0.948000; val_acc: 0.335000\n"
     ]
    }
   ],
   "source": [
    "# Train two identical nets, one with dropout and one without\n",
    "np.random.seed(231)\n",
    "num_train = 500\n",
    "small_data = {\n",
    "  'X_train': data['X_train'][:num_train],\n",
    "  'y_train': data['y_train'][:num_train],\n",
    "  'X_val': data['X_val'],\n",
    "  'y_val': data['y_val'],\n",
    "}\n",
    "\n",
    "solvers = {}\n",
    "dropout_choices = [1, 0.25]\n",
    "for dropout in dropout_choices:\n",
    "    model = FullyConnectedNet([500],dropout=dropout)\n",
    "    print(dropout)\n",
    "\n",
    "    solver = Solver(model, small_data,\n",
    "                  num_epochs=25, batch_size=100,\n",
    "                  update_rule='adam',\n",
    "                  optim_config={\n",
    "                    'learning_rate': 5e-4,\n",
    "                  },\n",
    "                  verbose=True, print_every=100)\n",
    "    solver.train()\n",
    "    solvers[dropout] = solver"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA4gAAAJNCAYAAABz4lt9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3XuclnWd8PHPN8Sc1AUPZHJQMFnMAKUdNdNVkxQtD2hFUml2Mksfd9tdNt2neIhnW12x7HF1t7XN1A4qGRGWSWWZ0cEYgsBDJKnFDJZ4gKxGQ/w+f9z34M04AwzMPdd9+Lxfr3ndc/2u3339vtfM5Xh/+Z0iM5EkSZIk6SVFByBJkiRJqg0miJIkSZIkwARRkiRJklRmgihJkiRJAkwQJUmSJEllJoiSJEmSJMAEUZIkImJQRPwxIvYrOhZJkooU7oMoSao3EfHHisOXAc8CG8vHH8jMLw18VJIk1T8TRElSXYuIR4D3ZeZ3t1Bnp8x8buCiGjiNfG+SpIHnEFNJUsOJiH+NiFsi4qaIeBp4Z0QcGRE/jYh1EfFoRFwVEYPL9XeKiIyI0eXjL5bPfysino6In0TEmF7aeklE3BoRvytf+66IeFXF+ZdFxJUR8duIWB8Rd0fES8vnjinHtD4iVkfE2eXyRRFxbsU13hcRd3WL9UMRsQr4Zbn86ohoj4g/RMTiiHhdxft3ioiPRcSvy+fbImJ4RPx3RPx7t/v5VkRcuOO/BUlSPTJBlCQ1qjOALwNDgFuA54C/A/YGjgJOAj6whfe/HfgYsCfwW+D/bqHuN4CxwCuAe4EvVJy7EpgIHFG+1r8Az5cTzm8CnwL2AiYBK/pwf6cBhwETysf3lNvZE7gV+EpXIgrMAN5C6Z6HAu8DngFuAN4eES8BiIh9gGOBm/sQhySpgZggSpIa1aLMvC0zn8/MzsxcnJn3ZOZzmfkQcC2lZKg3t2ZmW2ZuAL4EHNpTpfL1r8/MpzPzGWAW8DcRsWtEDALOBS7KzEczc2NmLipf853AHZk5txzT45m5rA/392+Z+VRmdpbj+EJmPlkebno58FfAgeW67wP+JTMfLMe7rFz3x0Bnxc9hOvDdzHy8D3FIkhqICaIkqVGtrjyIiIMi4pvloaB/AGZT6k3sze8qvv8zsFtPlcoroF4eEQ+Vr7uqfGpvYB9gZ+DXPbx1VC/l26r7/f1zRPwyItYDTwG78sL9bamtGyklq5Rfv9BLPUlSEzBBlCQ1qu6rsP03peGfB2bmXwEzgeiHds4B3ggcT2k4a1evXQC/B/4CvLKH963upRzgT5RWZ+3yih7qbLq/iHg98A/AmykNId0D+CMv3N+W2voCcGZETCrXua2XepKkJmCCKElqFrsD64E/lReR2dL8w75e91ngCUpJ3Se6TmTmRuB64NMR8Ypyb+NR5cVxvgicFBFvLi8is3dEHFJ+6zLgzRHREhF/DbxnG2J4DngcGExpmOuuFef/B/jXiHhllBwaEXuWY/xNub0bgK+Uh8lKkpqUCaIkqVn8I/Au4GlKvYm39NN1Pw+sKX/dB/y42/kPAw8AS4AngX+jtM3Uw8CpwEfK5T/nhQVnrqDUQ/gYcB2lZHJLbge+CzwIPAL8AXi04vwcYD5wZ/nctcAuFedvKLft8FJJanLugyhJUpOLiOOBzwEHpB8MJKmp2YMoSVITi4idKW3/8VmTQ0mSCaIkSU0qIiZQWvF0T+CqgsORJNUAh5hKkiRJkgB7ECVJkiRJZSaIkiRJkiQAdio6gIGw99575+jRo4sOQ5IkSZIKsWTJksczc9jW6jVFgjh69Gja2tqKDkOSJEmSChERv9mWeg4xlSRJkiQBJoiSJEmSpDITREmSJEkSYIIoSZIkSSozQZQkSZIkATWWIEbEdRHxWETc28v5iIirImJVRCyPiNcMdIySJEmS1KhqbZuL64GrgRt7OX8yMLb8dQTwX+VXSZIkSTVi/tIO5ixcyZp1nQwf2sKMKeOYOmlEw7ZbdNv9qaYSxMy8OyJGb6HK6cCNmZnATyNiaETsm5mPDkiAkiRJDaJRPsz2RbPdc5FJ2iXzVtC5YSMAHes6uWTeCoCqtl9Uu0W33d9qaojpNhgBrK44bi+XSZIkaRt1fZjtWNdJ8sKH2flLO4oOrWqKvOf5Szs46rLvMebib3LUZd8bsDaLut85C1duSpS6dG7YyJyFKxuy3aLb7m/1liBGD2XZY8WI8yKiLSLa1q5dW+WwJElSPSviA3yRGunD7LYq6p6LStSK/B2vWdfZp/J6b7fotvtbvSWI7cCoiuORwJqeKmbmtZnZmpmtw4YNG5DgJEnS9isqSWvG3rRG+jC7rYq656IStSJ/x8OHtvSpvN7bLbrt/lZvCeIC4JzyaqavBdY7/1CSpP7lcLjG701rpA+z26qoey4qUSvydzxjyjhaBg/arKxl8CBmTBnXkO0W3XZ/q6kEMSJuAn4CjIuI9oh4b0ScHxHnl6vcDjwErAI+C3yooFAlSWpIDofbenkjKPLDbFE9xUXdc1GJWpG/46mTRnDpmRMYMbSFAEYMbeHSMydUfbGWototuu3+VmurmE7fyvkELhigcCRJKlQRKxBuKVGrZttFD4fr6KGdRu5N6/pdDvTzVeRKj0Xd84wp4za7ZxiYRK2o+61sv4jkqKh2i267P9VUgihJkkqK+iBd5HC4opK0oj7AQ7HbLhTxYbaof4DoUsQ9F5moNUrCooFlgihJUg0q6oN0UYlakUlaM/amFaUZh/OCiZrqiwmiJElbUFQPT1EfpB0ON3CK7k0rQjMO55XqjQmiJEm9KLKHp6gP0g6HGzjN2JtWZE+xpG1jgihJUi+K7OEpeshlMyVqRWnG3rSie4olbZ0JoiRJvSiyh8cP0o2vWXvT/AcIqbaZIEqS1Iuie3j8IN3Y/EcASbXIBFGSpF40aw+PBo7/CCA1kOVz4c7ZsL4dhoyEyTNh4rSio+ozE0RJknphD48kaZssnwu3XQQbyqNO1q8uHUPdJYmRmUXHUHWtra3Z1tZWdBiSJEmSqqmoXrwrx5eSwu6GjIIP31v99rdBRCzJzNat1bMHUZIkSVL9K7IXb31738pr2EuKDkCSJElSg1k+t9SrNmto6XX53Oq3eefsF5LDLhs6S+XVNmRk38prmD2IklSH5i/tcF6cJKk2FdWTV2Qv3uSZm98zwOCWUnmdsQdRkurM/KUdXDJvBR3rOkmgY10nl8xbwfylHUWHVlXzl3Zw1GXfY8zF3+Soy77X8PcrSTusiF48KK4nr8hevInT4NSrSnMOidLrqVfV3QI1YA+iJNWdOQtXbrbtAkDnho3MWbiyYXsRu5LirvvuSoqBhr1nSQ2kiIVTmnE+XtG9eBOn1WVC2J09iJJUZ9b0sHH7lsobwZaSYkmqaV2J2vrVQL6QqFW7N68Z5+M1UC9ekexBlKQ6M3xoCx09JIPDh7ZUve2i5j42Y1IsqUFsKVFzPl7/a5BevCLZgyhJdWbGlHG0DB60WVnL4EHMmDKuqu0WOfext+R3IJJiSdohRSVqzsfTdjJBlKQ6M3XSCC49cwIjhrYQwIihLVx65oSq9+QVOcyzqKRYknZYUYna5JmlXrtKAz0f78P3wqx1pVeTw7pRc0NMI+Ik4P8Bg4D/yczLup3fH7gOGAY8CbwzM+tvB0pJda/IrSamThox4IuzFDnMs+te3dpD0g4pYrGYooZbdt3XQN+v6l5NJYgRMQi4BjgBaAcWR8SCzLy/otoVwI2ZeUNEHA9cCpw98NFKambNuKpmkXMfoZikWGpoRSRLRSpqVc8iEzXn42k71NoQ08OBVZn5UGb+BbgZOL1bnYOBO8vff7+H85JUdc24qqbDPKUGUtTKml1tN9PefOBwS9WVWksQRwCrK47by2WVfgG8ufz9GcDuEbHXAMQmSZs046qaRc19lFQFRSVLRSamRa7qKdWRmhpiCkQPZdnt+J+AqyPiXOBuoAN47kUXijgPOA9gv/32698oJTW9oodbFsVhnlI/K2qYZ1HJUlFbPkDp57t+dc/lkjaptR7EdmBUxfFIYE1lhcxck5lnZuYk4H+Xy9Z3v1BmXpuZrZnZOmzYsGrGLKkJOdxS0g4rsjetqJU1i96br8hVPaU6UWsJ4mJgbESMiYidgbOABZUVImLviOiK+xJKK5pK0oByuKXUYIqYF1fknLiikiX35pNqXk0NMc3M5yLiQmAhpW0ursvM+yJiNtCWmQuA44BLIyIpDTG9oLCAJTU1h1tKDaKo1S2L7E0ramXNorZ86OKqntJWRWb3KX6Np7W1Ndva2ooOQ5Ik1aIrx/cyN21UacXJRmu3aM22vYZUIyJiSWa2bq1eTfUgSpIkDbiievKK7k0rir14Uk2rtTmIkiRJA6uoeXHOiZNUg+xBlCRJza3Injx70yTVGHsQJUlSc7MnT5I2sQdRkiRtrqhFRIpcvMSePEkCTBAlSVKlorZ8KKpdSdJmHGIqSZJeUNTm7UVuGi9J2sQEUZIkvaCoLR+K3DRekrSJCaIkSbVq+dzSZuqzhpZel8+tfptFbflQVLuSpM2YIEqSVIu65uStXw3kC3Pyqp0kTp5Z2uKh0kBs+VBUu5KkzbhIjaS6N39pB3MWrmTNuk6GD21hxpRxTJ00ouiwpB2zpTl51Vy0pevaA72aaFHtSpI2Y4Ioqa7NX9rBJfNW0LlhIwAd6zq5ZN4KAJNE1bci5+QVteWDW01IUuEcYiqprs1ZuHJTctilc8NG5ixcWVBEUj9xTp4kqQAmiJLq2pp1nX0ql+qGc/IkSQUwQZRU14YPbelTuVQ3Jk6DU6+CIaOAKL2eepVDMCVJVeUcREl1bcaUcZvNQQRoGTyIGVPGFRiV1E+ckydJGmAmiJLqWtdCNK5iqqpZPteVNSVJTcMEUVLdmzpphAmhqqNrL8Ku7Sa69iIEk0RJUkNyDqIkSb3Z0l6EkiQ1oJpLECPipIhYGRGrIuLiHs7vFxHfj4ilEbE8It5YRJySpCZQ5F6EkiQVoKYSxIgYBFwDnAwcDEyPiIO7VfsoMDczJwFnAf85sFFKkpqGexFKkppMTSWIwOHAqsx8KDP/AtwMnN6tTgJ/Vf5+CLBmAOOTJDUT9yKUJDWZWlukZgSwuuK4HTiiW51ZwLcj4n8BuwJvGJjQJElNp2shGlcxlSQ1iVpLEKOHsux2PB24PjM/GRFHAl+IiPGZ+fxmF4o4DzgPYL/99qtKsJKkJuBehJKkJlJrQ0zbgVEVxyN58RDS9wJzATLzJ8AuwN7dL5SZ12Zma2a2Dhs2rErhSpIGzPK5cOV4mDW09Lp8btERSZLUcGotQVwMjI2IMRGxM6VFaBZ0q/NbYDJARLyKUoK4dkCjlCQNrK79CNevBvKF/QhNEiVJ6lc1lSBm5nPAhcBC4AFKq5XeFxGzI+K0crV/BN4fEb8AbgLOzczuw1AlSY3E/QglSRoQtTYHkcy8Hbi9W9nMiu/vB44a6LgkSQVyP0JJkgZEzSWIkurT/KUdzFm4kjXrOhk+tIUZU8YxddKIosNSoxgysjy8tIdySZLUb2pqiKmk+jR/aQeXzFtBx7pOEuhY18kl81Ywf2lH0aGpUbgfoSRJA8IEUdIOm7NwJZ0bNm5W1rlhI3MWriwoIlVNUSuJTpwGp14FQ0YBUXo99Sq3n5AkqZ85xFTSDluzrrNP5apTXSuJdi0W07WSKAxMouZ+hJIkVZ09iJJ22PChLX0qV51yJVFJkhqeCaKkHTZjyjhaBg/arKxl8CBmTBlXUESqClcSlSSp4TnEVNIO61qt1FVMG5wriUqS1PBMECX1i6mTRpgQNrrJMzefgwiuJCpJUoMxQZQajPsRqmq6Foi5c3ZpWOmQkaXk0IVjJElqGCaIUgPp2o+wa8uJrv0IAZNE9Q9XEpUkqaG5SI3UQNyPsIkUtR+hJElqaPYgSg3E/QibRNH7EUqSpIZlD6LUQNyPsEm4H6EkSaoSE0SpgbgfYZNwP0JJklQlDjGVGoj7ETYJ9yOUJElVYoIoNRj3I2wC7kcoSZKqxCGmklRvJk6DU6+CIaOAKL2eepUL1EiSpB1mD6Ik1SP3I5QkSVVgD6IkbS/3IpQkSQ2m5hLEiDgpIlZGxKqIuLiH81dGxLLy168iYl0RcUpqcl17Ea5fDeQLexGaJEqSpDpWUwliRAwCrgFOBg4GpkfEwZV1MvPDmXloZh4K/Acwb+AjldT03ItQkiQ1oFqbg3g4sCozHwKIiJuB04H7e6k/Hfg/AxSb1Cfzl3a43UQjcy9CSZLUgGqqBxEYAVRu7tVeLnuRiNgfGAN8bwDikvpk/tIOLpm3go51nSTQsa6TS+atYP7SjqJDU3/pbc9B9yKUJEl1rNYSxOihLHupexZwa2Zu7PFCEedFRFtEtK1du7bfApS2xZyFK+ncsPmj2blhI3MWriwoIvW7yTNLew9Wci9CSZJU52otQWwHRlUcjwTW9FL3LOCm3i6UmddmZmtmtg4bNqwfQ5S2bs26zj6Vqw65F6EkSWpAtTYHcTEwNiLGAB2UksC3d68UEeOAPYCfDGx40rYZPrSFjh6SweFDW3qorbrlXoSSJKnB1FQPYmY+B1wILAQeAOZm5n0RMTsiTquoOh24OTN7G34qFWrGlHG0DB60WVnL4EHMmDKuoIgkSZKkratKD2JEXAh8KTOf6ut7M/N24PZuZTO7Hc/aoQClKutardRVTCVJklRPqjXE9BXA4oj4OXAdsNDePjWbqZNGmBBKkiSprlRliGlmfhQYC3wOOBd4MCL+LSJeWY32JEmSJEk7rmpzEMs9hr8rfz1HaVGZWyPi8mq1KUmSJEnaftWag3gR8C7gceB/gBmZuSEiXgI8CPxzNdqVJEmSJG2/as1B3Bs4MzN/U1mYmc9HxClValOSJEmStAOqNcT0duDJroOI2D0ijgDIzAeq1KYkSZIkaQdUK0H8L+CPFcd/KpdJkiRJkmpUtRLEqNzWIjOfp3rDWSVJkiRJ/aBaCeJDEXFRRAwuf/0d8FCV2pLU7JbPhSvHw6yhpdflc4uOSJIkqS5VK0E8H3gd0AG0A0cA51WpLUnNbPlcuO0iWL8ayNLrbReZJEqSJG2Hqgz7zMzHgLOqcW1J2syds2FD5+ZlGzpL5ROnFROTJElSnarWPoi7AO8FXg3s0lWeme+pRnuSmtj69r6VS5IkqVfVGmL6BeAVwBTgB8BI4OkqtSWpmQ0Z2bdySZIk9apaCeKBmfkx4E+ZeQPwJmBCldqSVAuKWihm8kwY3LJ52eCWUrkkSZL6pFpbT2wov66LiPHA74DRVWpLUtG6ForpmgvYtVAMVH8eYNf175xdGlY6ZGQpOXT+oSRJUp9VK0G8NiL2AD4KLAB2Az5WpbYkFa3ohWImTjMhlCRJ6gf9niBGxEuAP2TmU8DdwAH93YakGuNCMZIkSQ2h3+cgZubzwIX9fV1JNcyFYiRJkhpCtRap+U5E/FNEjIqIPbu+qtSWpEpFLBbjQjGSJEkNoVoJ4nuACygNMV1S/mrbljdGxEkRsTIiVkXExb3UmRYR90fEfRHx5X6LWqp3XYvFrF8N5AuLxVQ7SZw4DU69CoaMAqL0eupVzguUJEmqM5GZRcewSUQMAn4FnAC0A4uB6Zl5f0WdscBc4PjMfCoiXp6Zj23puq2trdnWtk35qVTfrhxfTg67GTIKPnzvwMcjSZKkmhARSzKzdWv1qrKKaUSc01N5Zt64lbceDqzKzIfK17kZOB24v6LO+4FryovgsLXkUGoqLhYjSZKkHVCtbS4Oq/h+F2Ay8HNgawniCKCy+6MdOKJbnb8GiIgfAYOAWZl5xw5FKzWKISN76UF0sRhJkiRtXVUSxMz8X5XHETEE+MI2vDV6uly3452AscBxwEjghxExPjPXdWvzPOA8gP3222/bApfq3eSZm29YDy4WI0mSpG1WrUVquvszpaRua9qBURXHI4E1PdT5emZuyMyHgZU9XTszr83M1sxsHTZs2HaGLdUZF4uRJEnSDqjWHMTbeKHn7yXAwZQWltmaxcDYiBgDdABnAW/vVmc+MB24PiL2pjTk9KH+iFtqCBOnmRBKkiRpu1RrDuIVFd8/B/wmM7e6SkZmPhcRFwILKc0vvC4z74uI2UBbZi4onzsxIu4HNgIzMvOJ/r8FSZIkSWouVdnmotwD+GhmPlM+bgH2ycxH+r2xbeA2F5IkSZKaWaHbXABfAV5XcbyxXHZYz9XV6OYv7WDOwpWsWdfJ8KEtzJgyjqmTRhQdliRJkqQK1UoQd8rMv3QdZOZfImLnKrWlGjd/aQeXzFtB54aNAHSs6+SSeSsATBIlSZKkGlKtVUzXRsRpXQcRcTrweJXaUo2bs3DlpuSwS+eGjcxZuLKgiCRJkiT1pFo9iOcDX4qIq8vH7cA5VWpLNW7Nus4+lfenQoe2Lp8Ld86G9e2ljeonz3R1UUmSJNW0qiSImflr4LURsRulhXCerkY7qg/Dh7bQ0UMyOHxoS1XbLXRo6/K5m29Yv3516RhMEiVJklSzqjLENCL+LSKGZuYfM/PpiNgjIv61Gm2p9s2YMo6WwYM2K2sZPIgZU8ZVtd1Ch7beOfuF5LDLhs5SuSRJklSjqjUH8eTMXNd1kJlPAW+sUluqcVMnjeDSMycwYmgLAYwY2sKlZ06oei9ekUNbWd/Ltp+9lUuSJEk1oFpzEAdFxEsz81nYtA/iS6vUlurA1EkjBnzF0qKGtgKlOYfrV/dcLkmSJNWoavUgfhG4MyLeGxHvBb4D3FCltqQeFTW0FSgtSDO4WyI6uKVULkmSJNWoai1Sc3lELAfeAARwB7B/NdqSetPVY1nIKqZdC9G4iqkkSZLqSLWGmAL8DngemAY8DHy1im1JPSpiaOsmE6eZEEqSJKmu9GuCGBF/DZwFTAeeAG6htM3F6/uzHUmSJElS/+vvHsRfAj8ETs3MVQAR8eF+bkOSJEmSVAX9vUjNmykNLf1+RHw2IiZTmoMoSZIkSapx/ZogZubXMvNtwEHAXcCHgX0i4r8i4sT+bEuSJEmS1L+qss1FZv4pM7+UmacAI4FlwMXVaEuSJEmS1D+qtQ/iJpn5ZGb+d2YeX+22JEmSJEnbr+oJoiRJkiSpPpggSpIkSZKAGkwQI+KkiFgZEasi4kXzFiPi3IhYGxHLyl/vKyJOSZIkSWo0/b0P4g6JiEHANcAJQDuwOCIWZOb93arekpkXDniAqj/L58Kds2F9OwwZCZNnwsRpRUclSZIk1aRa60E8HFiVmQ9l5l+Am4HTC45J9Wr5XLjtIli/GsjS620XlcolSZIkvUitJYgjgNUVx+3lsu7eHBHLI+LWiBg1MKGp7tw5GzZ0bl62obNULkmSJOlFai1BjB7KstvxbcDozJwIfBe4occLRZwXEW0R0bZ27dp+DlN1YX1738olSZKkJldrCWI7UNkjOBJYU1khM5/IzGfLh58F/qanC2XmtZnZmpmtw4YNq0qwqnFDRvatXJIkSWpytZYgLgbGRsSYiNgZOAtYUFkhIvatODwNeGAA41M9mTwTBrdsXja4pVQuSZIk6UVqahXTzHwuIi4EFgKDgOsy876ImA20ZeYC4KKIOA14DngSOLewgFXbulYrdRVTSZIkaZtEZvcpfo2ntbU129raig5DkiRJkgoREUsys3Vr9WptiKkkSZIkqSAmiJIkSZIkwARRkiRJklRmgihJkiRJAkwQJUmSJEllJoiSJEmSJMAEUZIkSZJUZoIoSZIkSQJMECVJkiRJZSaIkiRJkiTABFGSJEmSVGaCKEmSJEkCTBAlSZIkSWUmiJIkSZIkAHYqOgANnPlLO5izcCVr1nUyfGgLM6aMY+qkEUWHJUmSJKlGmCA2iflLO7hk3go6N2wEoGNdJ5fMWwEwMEni8rlw52xY3w5DRsLkmTBxWvXblSRJkrTNHGLaJOYsXLkpOezSuWEjcxaurH7jy+fCbRfB+tVAll5vu6hULkmSJKlmmCA2iTXrOvtU3q/unA0burWzobNULkmSJKlmmCA2ieFDW/pU3q/Wt/etXJIkSVIhai5BjIiTImJlRKyKiIu3UO8tEZER0TqQ8dWrGVPG0TJ40GZlLYMHMWPKuOo3PmRk38olSZIkFaKmEsSIGARcA5wMHAxMj4iDe6i3O3ARcM/ARli/pk4awaVnTmDE0BYCGDG0hUvPnDAwC9RMngmDu/VUDm4plUuSJEmqGbW2iunhwKrMfAggIm4GTgfu71bv/wKXA/80sOHVt6mTRhSzrUXXaqWuYipJkiTVtFpLEEcAqyuO24EjKitExCRgVGZ+IyJMEOvFxGkmhJIkSVKNq6khpkD0UJabTka8BLgS+MetXijivIhoi4i2tWvX9mOIkiRJktSYai1BbAdGVRyPBNZUHO8OjAfuiohHgNcCC3paqCYzr83M1sxsHTZsWBVDliRJkqTGUGsJ4mJgbESMiYidgbOABV0nM3N9Zu6dmaMzczTwU+C0zGwrJlxJkiRJahw1lSBm5nPAhcBC4AFgbmbeFxGzI+K0YqOTJEmSpMZWa4vUkJm3A7d3K+txP4TMPG4gYmoYy+e6kqgkSZKkXtVcgqgqWT4XbrsINnSWjtevLh2DSaIkSZIkoMaGmKqK7pz9QnLYZUNnqVySJEmSMEFsHuvb+1YuSZIkqemYIDaLISP7Vi5JkiSp6ZggNovJM2Fwy+Zlg1tK5ZIkSZKECWLzmDgNTr0KhowCovR66lUuUCNJkiRpE1cxbSYTp5kQSpIkSeqVPYiSJEmSJMAEUZIkSZJUZoIoSZIkSQJMECVJkiRJZSaIkiRJkiTABFGSJEmSVGaCKEmSJEkCTBAlSZIkSWUmiJIkSZIkwARRkiRJklRmgihJkiRJAkwQJUmSJEllNZcgRsRJEbEyIlZFxMU9nD8/IlZExLKIWBQRBxcRpyRJkiQ1mppKECNiEHANcDJwMDC9hwTwy5k5ITMPBS4HPjXAYUqSJElSQ6qpBBE4HFiVmQ9l5l+Am4HTKytk5h8qDncFcgDjkyRJkqSGtVPRAXQzAlhdcdwOHNG9UkRcAPwDsDNw/MCEJkmSJEmNrdZ6EKOHshf1EGZOnHlqAAAgAElEQVTmNZn5SuAjwEd7vFDEeRHRFhFta9eu7ecwJUmSJKnx1FqC2A6MqjgeCazZQv2bgak9ncjMazOzNTNbhw0b1o8hSpIkSVJjqrUEcTEwNiLGRMTOwFnAgsoKETG24vBNwIMDGJ8kSZIkNayamoOYmc9FxIXAQmAQcF1m3hcRs4G2zFwAXBgRbwA2AE8B7youYkmSJElqHDWVIAJk5u3A7d3KZlZ8/3cDHpQkSZIkNYFaG2IqSZIkSSpIzfUgNoP5SzuYs3Ala9Z1MnxoCzOmjGPqpBFFhyVJkiSpyZkgDrD5Szu4ZN4KOjdsBKBjXSeXzFsBYJIoSZIkqVAOMR1gcxau3JQcduncsJE5C1cWFJEkSZIklZggDrA16zr7VC5JkiRJA8UhpgNs+NAWOnpIBocPbSkgGkmSpOJt2LCB9vZ2nnnmmaJDkereLrvswsiRIxk8ePB2vd8EcYDNmDJuszmIAC2DBzFjyrgCo5IkSSpOe3s7u+++O6NHjyYiig5HqluZyRNPPEF7eztjxozZrms4xHSATZ00gkvPnMCIoS0EMGJoC5eeOcEFaiRJUtN65pln2GuvvUwOpR0UEey111471BtvD2IBpk4aYUIoSZJUweRQ6h87+t+SPYiSJElqeu95z3t4+ctfzvjx43utk5lcdNFFHHjggUycOJGf//znm87dcMMNjB07lrFjx3LDDTdstb277rqLU045pV9i3x7XX389a9asKaz9/nbHHXcwbtw4DjzwQC677LIe63zqU5/i4IMPZuLEiUyePJnf/OY3m84NGjSIQw89lEMPPZTTTjttq+098sgjW3xWqm3+/Pncf//9Vbm2CaIkSZKa3rnnnssdd9yxxTrf+ta3ePDBB3nwwQe59tpr+eAHPwjAk08+ycc//nHuuecefvazn/Hxj3+cp556arvieO6557brfX3VSAnixo0bueCCC/jWt77F/fffz0033dRj8jRp0iTa2tpYvnw5b3nLW/jnf/7nTedaWlpYtmwZy5YtY8GCBdsdy0D9/kwQJUmSpLL5Szs46rLvMebib3LUZd9j/tKOHb7mMcccw5577rnFOl//+tc555xziAhe+9rXsm7dOh599FEWLlzICSecwJ577skee+zBCSec0GOyeccdd3DQQQdx9NFHM2/evE3ls2bN4rzzzuPEE0/knHPO4ZlnnuHd7343EyZMYNKkSXz/+98HSknd6aefzkknncS4ceP4+Mc/vukan/rUpxg/fjzjx4/n05/+NPDiXq4rrriCWbNmceutt9LW1sY73vEODj30UDo7B3i7teVz4crxMGto6XX53B263M9+9jMOPPBADjjgAHbeeWfOOussvv71r7+o3utf/3pe9rKXAfDa176W9vb2PrWzZMkSDjnkEI488kiuueaaTeXXX389b33rWzn11FM58cQTyUxmzJjB+PHjmTBhArfccgtQ6jU+5phjOOOMMzj44IM5//zzef755wG46aabmDBhAuPHj+cjH/nIpmvvtttum76/9dZbOffcc/nxj3/MggULmDFjBoceeii//vWv+3QfW+McREmSJNWN+Us7NlsRvmNdJ5fMWwFQ9TUeOjo6GDVq1KbjkSNH0tHR0Wt5pWeeeYb3v//9fO973+PAAw/kbW9722bnlyxZwqJFi2hpaeGTn/wkACtWrOCXv/wlJ554Ir/61a+AUjJ077338rKXvYzDDjuMN73pTUQEn//857nnnnvITI444giOPfZY9thjjx7v4y1veQtXX301V1xxBa2trf3ys9lmy+fCbRfBhnJSun516Rhg4rTtumRPP/977rlni+/53Oc+x8knn7zp+JlnnqG1tZWddtqJiy++mKlTp77oPe9+97v5j//4D4499lhmzJix2bmf/OQnLF++nD333JOvfvWrLFu2jF/84hc8/vjjHHbYYRxzzDFA6fd3//33s//++3PSSScxb948Xve61/GRj3yEJUuWsMcee3DiiScyf/78HmMAeN3rXsdpp53GKaecwlve8pZt/jltK3sQJUmSVDfmLFy52XZhAJ0bNjJn4cqqt52ZLyqLiF7LK/3yl79kzJgxjB07lojgne9852bnTzvtNFpaSvtiL1q0iLPPPhuAgw46iP33339TgnjCCSew11570dLSwplnnsmiRYtYtGgRZ5xxBrvuuiu77bYbZ555Jj/84Q/75Z773Z2zX0gOu2zoLJVvp235+Vf64he/SFtb22ZJ3m9/+1va2tr48pe/zN///d+/qFdu/fr1rFu3jmOPPRZg0++nS1cPMpR+f9OnT2fQoEHss88+HHvssSxevBiAww8/nAMOOIBBgwYxffp0Fi1axOLFiznuuOMYNmwYO+20E+94xzu4++67t++H0Q9MEIvQz93qkiRJzWLNup6HQ/ZW3p9GjhzJ6tWrNx23t7czfPjwXsu721LSsuuuu276vqeEp7dr9JagAuy0006bhjACO7T1Qb9Z38uwzt7Kt8G2/vwBvvvd7/KJT3yCBQsW8NKXvnRTeVf9Aw44gOOOO46lS5du9r7MHPDfX/f6A/X7M0EcaF3d6utXA/lCt7pJoiRJ0lYNH9rSp/L+dNppp3HjjTeSmfz0pz9lyJAh7LvvvkyZMoVvf/vbPPXUUzz11FN8+9vfZsqUKZu996CDDuLhhx/e1DN100039drOMcccw5e+9CUAfvWrX/Hb3/6WcePGAfCd73yHJ598ks7OTubPn89RRx3FMcccw/z58/nzn//Mn/70J772ta/xt3/7t+yzzz489thjPPHEEzz77LN84xvf2NTG7rvvztNPP93fP6KtGzKyb+Xb4LDDDuPBBx/k4Ycf5i9/+Qs333xzjyuRLl26lA984AMsWLCAl7/85ZvKn3rqKZ599lkAHn/8cX70ox9x8MEHb/beoUOHMmTIEBYtWgSw6ffTk2OOOYZbbrmFjRs3snbtWu6++24OP/xwoDTE9OGHH+b555/nlltu4eijj+aII47gBz/4AY8//jgbN27kpptu2tRTuc8++/DAAw/w/PPP87WvfW1TG9X8/ZkgDrQqdKtLkiQ1ixlTxtEyeNBmZS2DBzFjyrgduu706dM58sgjWblyJSNHjuRzn/scAJ/5zGf4zGc+A8Ab3/hGDjjgAA488EDe//7385//+Z8A7LnnnnzsYx/jsMMO47DDDmPmzJkvWvBml1124dprr+VNb3oTRx99NPvvv3+vsXzoQx9i48aNTJgwgbe97W1cf/31m3q7jj76aM4++2wOPfRQ3vzmN9Pa2sprXvMazj33XA4//HCOOOII3ve+9zFp0iQGDx7MzJkzOeKIIzjllFM46KCDNrVx7rnncv755w/8IjWTZ8Lgbsn84JZS+XbaaaeduPrqq5kyZQqvetWrmDZtGq9+9asBmDlz5qZVSWfMmMEf//hH3vrWt262ncUDDzxAa2srhxxyCK9//eu5+OKLX5QgAnz+85/nggsu4Mgjj9w0HLgnZ5xxBhMnTuSQQw7h+OOP5/LLL+cVr3gFAEceeSQXX3wx48ePZ8yYMZxxxhnsu+++XHrppbz+9a/nkEMO4TWveQ2nn346AJdddhmnnHIKxx9/PPvuu++mNs466yzmzJnDpEmT+n2RmthSl2ajaG1tzba2tqLDKJk1FOjpZx4wa91ARyNJklS4Bx54gFe96lXbXH/+0g7mLFzJmnWdDB/awowp46q+QE0tuP7662lra+Pqq68uOpQds3xuqXNkfXup53DyzO1eoKae3HXXXVxxxRWb9eRWS0//TUXEkszc6qpErmI60IaMLA8v7aFckiRJWzV10oimSAgb1sRpTZEQ1quaG2IaESdFxMqIWBURF/dw/h8i4v6IWB4Rd0ZE7/3ztagK3eqSJElqfOeee2799x42seOOO25Aeg93VE0liBExCLgGOBk4GJgeEd0HAC8FWjNzInArcPnARrmDJk6DU6+CIaOAKL2eepX/iiJJkiSpcLU2xPRwYFVmPgQQETcDpwP3d1XIzO9X1P8psPkmMvXAbnVJkqTNbG0bAUnbZkfXmKmpHkRgBFA5Qa+9XNab9wLfqmpEkiRJqqpddtmFJ554Yoc/2ErNLjN54okn2GWXXbb7GrXWg9jTPxv1+JciIt4JtALH9nL+POA8gP3226+/4pMkSVI/GzlyJO3t7axdu7boUKS6t8suuzBy5PYvgFlrCWI7MKrieCSwpnuliHgD8L+BYzPz2Z4ulJnXAtdCaZuL/g9VkiRJ/WHw4MGMGTOm6DAkUXtDTBcDYyNiTETsDJwFLKisEBGTgP8GTsvMxwqIUZIkSZIaUk0liJn5HHAhsBB4AJibmfdFxOyIOK1cbQ6wG/CViFgWEQt6uZwkSZIkqQ9qbYgpmXk7cHu3spkV379hwIOSJEmSpCYQzbBaVESsBX5TdBw92Bt4vOgg1LB8vlRNPl+qJp8vVZvPmKqpVp+v/TNz2NYqNUWCWKsioi0zW4uOQ43J50vV5POlavL5UrX5jKma6v35qqk5iJIkSZKk4pggSpIkSZIAE8SiXVt0AGpoPl+qJp8vVZPPl6rNZ0zVVNfPl3MQJUmSJEmAPYiSJEmSpDITxAJExEkRsTIiVkXExUXHo8YTEY9ExIqIWBYRbUXHo/oWEddFxGMRcW9F2Z4R8Z2IeLD8ukeRMap+9fJ8zYqIjvLfsGUR8cYiY1T9iohREfH9iHggIu6LiL8rl/s3TDtsC89XXf8Nc4jpAIuIQcCvgBOAdmAxMD0z7y80MDWUiHgEaM3MWtyDR3UmIo4B/gjcmJnjy2WXA09m5mXlf+jaIzM/UmScqk+9PF+zgD9m5hVFxqb6FxH7Avtm5s8jYndgCTAVOBf/hmkHbeH5mkYd/w2zB3HgHQ6sysyHMvMvwM3A6QXHJEm9ysy7gSe7FZ8O3FD+/gZK/0OU+qyX50vqF5n5aGb+vPz908ADwAj8G6Z+sIXnq66ZIA68EcDqiuN2GuBBUs1J4NsRsSQizis6GDWkfTLzUSj9DxJ4ecHxqPFcGBHLy0NQHf6nHRYRo4FJwD34N0z9rNvzBXX8N8wEceBFD2WO81V/OyozXwOcDFxQHsIlSfXiv4BXAocCjwKfLDYc1buI2A34KvD3mfmHouNRY+nh+arrv2EmiAOvHRhVcTwSWFNQLGpQmbmm/PoY8DVKQ5ul/vT78tyLrjkYjxUcjxpIZv4+Mzdm5vPAZ/FvmHZARAym9OH9S5k5r1zs3zD1i56er3r/G2aCOPAWA2MjYkxE7AycBSwoOCY1kIjYtTxRmojYFTgRuHfL75L6bAHwrvL37wK+XmAsajBdH9zLzsC/YdpOERHA54AHMvNTFaf8G6Yd1tvzVe9/w1zFtADlpW4/DQwCrsvMTxQckhpIRBxAqdcQYCfgyz5j2hERcRNwHLA38Hvg/wDzgbnAfsBvgbdmpguNqM96eb6OozQ0K4FHgA90zReT+iIijgZ+CKwAni8X/wuleWL+DdMO2cLzNZ06/htmgihJkiRJAhxiKkmSJEkqM0GUJEmSJAEmiJIkSZKkMhNESZIkSRJggihJkiRJKjNBlCRpG0XExohYVvF1cT9ee3RE1NVeWZKkxrNT0QFIklRHOjPz0KKDkCSpWuxBlCRpB0XEIxHx7xHxs/LXgeXy/SPizohYXn7dr1y+T0R8LSJ+Uf56XflSgyLisxFxX0R8OyJaCrspSVJTMkGUJGnbtXQbYvq2inN/yMzDgauBT5fLrgZuzMyJwJeAq8rlVwE/yMxDgNcA95XLxwLXZOargXXAm6t8P5IkbSYys+gYJEmqCxHxx8zcrYfyR4DjM/OhiBgM/C4z94qIx4F9M3NDufzRzNw7ItYCIzPz2YprjAa+k5ljy8cfAQZn5r9W/84kSSqxB1GSpP6RvXzfW52ePFvx/UZcK0CSNMBMECVJ6h9vq3j9Sfn7HwNnlb9/B7Co/P2dwAcBImJQRPzVQAUpSdKW+C+TkiRtu5aIWFZxfEdmdm118dKIuIfSP75OL5ddBFwXETOAtcC7y+V/B1wbEe+l1FP4QeDRqkcvSdJWOAdRkqQdVJ6D2JqZjxcdiyRJO8IhppIkSZIkwB5ESZIkSVKZPYiSJJVFxOiIyIhwjr4kqSmZIEqSGkZELIyI2T2Unx4RvzPxkyRpy0wQJUmN5Hrg7IiIbuVnA1/KzOcGPqT+ESX+f1uSVFX+j0aS1EjmA3sCf9tVEBF7AKcAN5aP3xQRSyPiDxGxOiJmbevFI+LiiPh1RDwdEfdHxBndzr8/Ih6oOP+acvmoiJgXEWsj4omIuLpcPisivljx/s2GuEbEXRHxiYj4EfBn4ICIeHdFGw9FxAe6xXB6RCwr39+vI+KkiHhrRCzpVu8fI2L+tt67JKk5mCBKkhpGZnYCc4FzKoqnAb/MzF+Uj/9UPj8UeBPwwYiYuo1N/JpS8jkE+DjwxYjYFyAi3grMKl/7r4DTgCciYhDwDeA3wGhgBHBzH27rbOA8YPfyNR6jlPD+FaV9Fa+sSEQPp5QIzyjf3zHAI8ACYExEvKriuu8EvtCHOCRJTcAEUZLUaG4A3hoRLeXjc8plAGTmXZm5IjOfz8zlwE3Asdty4cz8SmauKb/3FuBB4PDy6fcBl2fm4ixZlZm/KZ8fDszIzD9l5jOZuagP93N9Zt6Xmc9l5obM/GZm/rrcxg+Ab/NCj+l7gesy8zvlGDsy85eZ+SxwC6WkkIh4NaVk9Rt9iEOS1ARMECVJDaWcfK0FTo+IA4DDgC93nY+IIyLi++XhnuuB84G9t+XaEXFOefjmuohYB4yveO8oSj2M3Y0CfrMD8x9Xd4vh5Ij4aUQ8WY7hjdsQA5SS5LeX52eeDcwtJ46SJG1igihJakQ3Uuo5PBv4dmb+vuLclykNuRyVmUOAzwDdF7V5kYjYH/gscCGwV2YOBe6teO9q4JU9vHU1sF8vK6j+CXhZxfEreqizacPiiHgp8FXgCmCfcgy3b0MMZOZPgb9Q6m18Ow4vlST1wARRktSIbgTeALyfiuGlZbsDT2bmM+U5e2/fxmvuSilZWwsQEe+m1IPY5X+Af4qIvymvOHpgOan8GfAocFlE7BoRu0TEUeX3LAOOiYj9ImIIcMlWYtgZeGk5huci4mTgxIrznwPeHRGTI+IlETEiIg6qOH8jcDXwXB+HuUqSmoQJoiSp4WTmI8CPKSV1C7qd/hAwOyKeBmZSWtRmW655P/BJ4CfA74EJwI8qzn8F+ASlHsqnKa+ompkbgVOBA4HfAu3A28rv+Q6luYHLgSVsZU5gZj4NXFSO+SlKye2CivM/o7xwDbAe+AGwf8UlvkApqbX3UJLUo8jMrdeSJEl1r7xwz2PAazLzwaLjkSTVHnsQJUlqHh8EFpscSpJ609OEeUmS1GAi4hFKi9ls656PkqQm5BBTSZIkSRLgEFNJkiRJUpkJoiRJkiQJaJI5iHvvvXeOHj266DAkSZIkqRBLlix5PDOHba1eUySIo0ePpq2tregwJEmSJKkQEfGbbannEFNJkiRJEmCCKEmSJEkqM0GUJEmSJAEmiJIkSZKkMhNESZIkSRJggihJkiRJKjNBlCRJakbL58KV42HW0NLr8rlFRySpBjTFPoiSJEmqsHwu3HYRbOgsHa9fXToGmDituLgkFc4eREmSpGZz5+wXksMuGzpL5ZKamgmiJElSs1nf3rdySU3DBFGSJKnZDBnZt3JJW9cg83pNECVJkhrkg902mzwTBrdsXja4pVRebc32s1Zz6JrXu341kC/M663D59sEUZIkNbcG+mC3zSZOg1OvgiGjgCi9nnpV9ReoacaftZpDA83rdRVTSZLU3Lb0wa6RV/ScOG3g769Zf9ZqfA00r9ceREmS1Nwa6INdzfNnrUbVQPN6TRAlSVJza6APdjWv6J+18x9VLUXO6+1nJoiSJKm5NdAHu5pX9OI4zn9UtRQ1r7cKnIMoSZI2t3xuaU7Y+vZSz87kmXX5IWebdd1bM91zUYr8WTv/UdVWxLzeKjBBlCRJL+jqZen6IN3VywIN8cGnVw3ywa4uFPWzdv5jc2i2f+CqAoeYSpKkFzTQUu3SZoqe/6jqcxhxvzBBlCRJL7CXRY3KuaaNz3/g6hcmiJIk6QX2sqhRNdAiIuqF/8DVL6qaIEbESRGxMiJWRcTFPZw/PyJWRMSyiFgUEQeXy0+IiCXlc0si4viK99xVvuay8tfLq3kPkiQ1laJXmWy2LQia8Z6LNHEafPhemLWu9GpyWD1FPNv+A1e/qFqCGBGDgGuAk4GDgeldCWCFL2fmhMw8FLgc+FS5/HHg1MycALwL+EK3970jMw8tfz1WrXuQJKnpFNXL0oxzh5rxntUcinq2HUbcL6q5iunhwKrMfAggIm4GTgfu76qQmX+oqL8rkOXypRXl9wG7RMRLM/PZKsYrSZKgmFUmm3ELgma8ZzWHop5tt6zpF9VMEEcAqyuO24EjuleKiAuAfwB2Bo7vfh54M7C0W3L4+YjYCHwV+NfMzH6LWpIkDbxmnDvUjPes5lDks+2WNTusmnMQo4eyFyVymXlNZr4S+Ajw0c0uEPFq4N+BD1QUv6M89PRvy19n99h4xHkR0RYRbWvXrt3OW5AkSQOiGecONeM9qzn4bNe1aiaI7cCoiuORwJot1L8ZmNp1EBEjga8B52Tmr7vKM7Oj/Po08GVKQ1lfJDOvzczWzGwdNmzYdt+EJEkaAM04d6gZ77lZNdtiRD7bda2aCeJiYGxEjImInYGzgAWVFSJibMXhm4AHy+VDgW8Cl2Tmjyrq7xQRe5e/HwycAtxbxXuQJEkDoRm3IGjGe25GzbgYkc92XYtqTt+LiDcCnwYGAddl5iciYjbQlpkLIuL/AW8ANgBPARdm5n0R8VHgEsoJY9mJwJ+Au4HB5Wt+F/iHzNy4pThaW1uzra2tn+9OkiRJ2oorx5eTw26GjCpttSENkIhYkpmtW63XDOu7mCBKkiSpELOG0sMyHECU9mOUBsi2JojVHGIqSZIkNTcXbFGdMUGUJNWHZlvkQVJjcMEW1Zlq7oMoSVL/6FrkoWvj5f/f3r3H2VXX9/5/fc4QyFQsAUSEBCFIThCTkHiGAEK5CgELISAiqReCVsTiD1t/nRJObQr8DoeUWPXwkKpYFGy5RU4IoQpBCV6wXDKYmAg0goAwEyzhEqoyYAif3x97z7gzzExmkll7Zfa8no/Hfuy9vuuyv2tmZWe/53tZXZM8gJMeSNq2efN2DTMGREnSwK1aWM6XnLsu+UM47LKhs1LulyxJ27qybt5e1me2hjUDoiRpYMpsxXupfXDlkjTS2fNCW8gxiJKkgemvFa9oTvIgSYNT5me2hjUDoiRpYMpsxXOSB0kaHHteaAsZECVJA1NmK96UM+DkKyo3liYqzydfYTcpSeqLPS+0hRyDKEkamGPnbTqeBerbilfWJA+SNByV/ZmtYcsWREnSwNiKV3/e+1HSlvIzW1soMrPsOhSupaUl29rayq6GJEkD13MGQqj89d8veJKkLRARD2Zmy+a2swVRkqRtkTMQSpJK4BhESZK2Rc5AqIItXtHBgqVrWLu+kz3HNNM6YyKzpo0tu1qSSmYLoiQNR45Na3zOQKgCLV7RwYWLVtOxvpMEOtZ3cuGi1Sxe0VF21SSVzIAoScNN19i0l54GsvJ82/mGxEbjvR9VoAVL19C5YeMmZZ0bNrJg6ZqSaiRpW2FAlKThxrFpI4MzEKpAa9d3Dqpc0sjhGERJGm4cmzZyeO9HFWTPMc109BIG9xzT3MvWQ8uxj2pUjXJt24IoScONY9Pqy/GeakCtMybSPKppk7LmUU20zphY6Ps69rG+Fq/o4LD5yxg/9zscNn+ZP+cCNdK1bUCUpOHGsWn143hPNahZ08Zy2WmTGTummQDGjmnmstMmF97a4djH+mmkwDIcNNK1bRdTSRpuuroc3nVJpVvpTuMq4dCuiEOvv/Ge/rw1zM2aNrbu3d8c+1g//QWW4djtcaDK6ubZSNe2AVGShiPHptWH4z2lIVXm2McylRFayg4sZZxzV6tpVzDuajUFCn/vRrq27WIqSVJfHO8pDamyxj6Wqayunn0Fk3pNRFTGOZfZzbORrm0DoiRJfXG8pzSkyhr7WKayQkuZgaWscy6z1bSRrm27mEqS1BfHe0pDroyxj11GUlfPrvMaSePxyu7mWea1PZQKDYgRcQLwf4Am4J8zc36P9ecC5wEbgd8C52Tmw9V1FwIfr647PzOXDuSYkiQNKcd7Sg2hrPFpZYaWsgJLWefcOmPiJr9jGL7dPMtUWBfTiGgCrgROBA4AZkfEAT02uz4zJ2fmVOBy4AvVfQ8AzgTeBZwA/FNENA3wmNoGeR8eSZJUppHY1bMsZZ1zI3XzLFORLYjTgccy83GAiLgROAV4uGuDzPyvmu3fBGT19SnAjZn5KvBERDxWPR6bO6a2PWXOKCVJkgQjs6tnWco850bp5lmmIgPiWODpmuV24OCeG0XEecBnge2BY2r2va/Hvl2/6c0eU9uWkXofHkmStO0YiV09yzQSz7lRFDmLafRSlm8oyLwyM98BXAB8bjP7DuiYABFxTkS0RUTbunXrBlhlFaHs+/BIkganrGEBDkdQkUZiV09pSxTZgtgO7FWzPA5Y28/2NwJfGcC+AzpmZl4FXAXQ0tLSa4hUfZQ9o9RIU8YMbaVbtdBZJqUhUtawAIcjqGgjsauntCWKDIjLgQkRMR7ooDLpzJ/VbhAREzLz0erinwJdr5cA10fEF4A9gQnAA1RaEPs9prY9zihVPyPyC9aqhXDb+bCh+keIl56uLIMhUdoCZQ0LcDiC6sFuj9LmFdbFNDNfAz4NLAUeARZm5kMRcUlEzKxu9umIeCgiVlIZh3hWdd+HgIVUJp+5AzgvMzf2dcyizkFDwxml6qesGdpKddclf8GqEIEAACAASURBVAiHXTZ0VsolDVpZwwIcjiBJ24ZC74OYmd8FvtujbF7N68/0s++lwKUDOaa2ff7Frj5G5Besl9oHVy6pX2UNC3A4giRtG4qcpEZSnfX1Raqhv2DtNG5w5ZL6VdZEHmVPIOLEPJJUYUCUGkjZX7BKcew8GNUjAI9qrpRLGrSyhgWUORyha/x2x/pOkj+M3y46rJX1vpLUn8hs/Ak+W1pasq2trexqSHXhLKbOYippcA6bv6zX7q1jxzTzk7nH9LLH8H5fSSNTRDyYmS2b267QMYiS6m9EjveccoaBUNIWc2IeSfoDu5hKkqQRrazx2yNy3LikbZ4BUZIkjWgjdWIeSeqNXUwlSdpGjcgxxSXo+pnW+2dd1vtKUn+cpEaSpG1Q1wyXnRs2dpc1j2qq28yekqTG4iQ1kiQNgbJa8RYsXbNJOATo3LCRBUvXGBAlSYUxIEqS1IeerXhd96kDCg9pznApSSqDk9RIRVm1EL44CS4aU3letbDsGkkapP5a8YrmDJeSpDIYEKUirFoIt50PLz0NZOX5tvMNidIwU2YrnjNcSpLKYECUinDXJbChxxfIDZ2VcknDRpmteLOmjeWy0yYzdkwzAYwd0+wENZKkwjkGUSrCS+2DK5e0TWqdMbHXmUTr1Yo3a9pYA6Ekqa5sQZSKsNO4wZUPpbLGPjrmUg3IVjxJ0khjC6JUhGPnVcYc1nYzHdVcKS9S19jHrvftGvsIMOWMxnvfsq1aWOk2/FJ7JfwfO6+xz3eEshVPkjSS2IKoxlZWq9aUM+DkK2CnvYCoPJ98RfHhoayxjyNxzKUTEUmSpAZkC6IaV9mtWlPOqH9rUlljH0fimMv+QrGtiJIkaZiyBVGNayS2apU19rHMMZdlGYmhuGSLV3Rw2PxljJ/7HQ6bv4zFKzrKrpIkSQ3HgKjGNRK/wB87rzLWsVY9xj6W9b5lGomhuESLV3Rw4aLVdKzvJIGO9Z1cuGi1IVGSpCFmQFTjGolf4Msa+1jW+5ap5FA80lrTFixds8mtJgA6N2xkwdI1JdVIkqTG5BhENa6yZhItWxljH8t837J0nWsJs5h2taZ1Baau1jSgYWfbXLu+c1DlkiRpyxgQ1bhK/AKvEaKkUNxfa1qjBsQ9xzTT0UsY3HNMcy9bS5KkLWVAVGMrsVVr8YoOFixdw9r1new5ppnWGRMb9su76msktqa1zpi4SaspQPOoJlpnTCyxVpIkNR4DolSAkdgFUPUzElvTuv7d+EcXSZKKVWhAjIgTgP8DNAH/nJnze6z/LPDnwGvAOuBjmfmriDga+GLNpvsDZ2bm4oi4BjgSeKm6bk5mrizyPKTBGoldAFU/I7U1bda0sf77kSSpYIUFxIhoAq4EjgPageURsSQzH67ZbAXQkpkvR8SngMuBD2bm3cDU6nF2AR4D7qzZrzUzby6q7tLWGoldAFU/tqZJkqSiFNmCOB14LDMfB4iIG4FTgO6AWA2CXe4DPtzLcU4Hbs/MlwusqzSkRmIXQNWXrWmSJKkIRd4HcSzwdM1ye7WsLx8Hbu+l/Ezghh5ll0bEqoj4YkTssHXVlIZe64yJNI9q2qRsJHQBlCRJ0vBWZECMXsqy1w0jPgy0AAt6lO8BTAaW1hRfSGVM4kHALsAFfRzznIhoi4i2devWDb720laYNW0sl502mbFjmglg7JhmLjttsi0+kiRJ2qYV2cW0HdirZnkcsLbnRhHxXuBvgSMz89Ueq88AbsnMDV0FmflM9eWrEfFN4K97e/PMvAq4CqClpaXXYCoVyS6AkiRJGm6KDIjLgQkRMR7ooNJV9M9qN4iIacDXgBMy89lejjGbSoth7T57ZOYzERHALODnRVRe0uB430dJkqThr7CAmJmvRcSnqXQPbQK+kZkPRcQlQFtmLqHSpXRH4NuVvMdTmTkTICL2odIC+cMeh74uInaj0oV1JXBuUecgaWC876MkSVJjiMz+e19WQ951mflifao09FpaWrKtra3sakgN67D5y3qdtXXsmGZ+MveYEmokSZKkWhHxYGa2bG67gUxS8zYq9zBcGBEnVLt2SlI37/soSZLUGDYbEDPzc8AE4GpgDvBoRPzviHhHwXWTNEz0dX9H7/soSZI0vAzoNhdZ6Yf66+rjNWBn4OaIuLzAukkaJrzvoyRJUmPY7CQ1EXE+cBbwHPDPQGtmboiI/wY8CvxNsVWUtK3rmojGWUwlSZKGt4HMYvoW4LTM/FVtYWa+HhEnFVMtScON932UJEka/gbSxfS7wAtdCxHx5og4GCAzHymqYpIkSZKk+hpIQPwK8Nua5d9VyyRJkiRJDWQgATGy5maJmfk6A+uaKkmSJEkaRgYSEB+PiPMjYlT18Rng8aIrJkmSJEmqr4EExHOB9wAdQDtwMHBOkZWSJEmSJNXfZruKZuazwJl1qIskSZIkqUQDuQ/iaODjwLuA0V3lmfmxAuslSZIkSaqzgXQx/RfgbcAM4IfAOOA3RVZKkiRJklR/A5mNdL/M/EBEnJKZ10bE9cDSoismSdu6xSs6WLB0DWvXd7LnmGZaZ0xk1rSxZVdLkiRpiw0kIG6oPq+PiEnAr4F9CquRJA0Di1d0cOGi1XRu2AhAx/pOLly0GsCQKEmShq2BdDG9KiJ2Bj4HLAEeBv6h0FpJ0jZuwdI13eGwS+eGjSxYuqakGkmSJG29flsQI+K/Af+VmS8CPwL2rUutJGkbt3Z956DKJUmShoN+WxAz83Xg03WqiyQNG3uOaR5UuSRJ0nAwkC6m34uIv46IvSJil65H4TXTkFu8ooPD5i9j/NzvcNj8ZSxe0VF2laRhq3XGRJpHNW1S1jyqidYZE0uqkSRJ0tYbyCQ1Xfc7PK+mLLG76bDihBrS0Or6d+MsppIkqZFsNiBm5vh6VETF6m9CDb/QSltm1rSx/vuRJEkNZbMBMSI+2lt5Zn5r6KujojihhiRJkqTNGUgX04NqXo8GjgV+ChgQh5E9xzTT0UsYdEINSZIkSV0G0sX0/6ldjoidgH8prEYqROuMiZuMQQQn1JAkSZK0qYHMYtrTy8CEgWwYESdExJqIeCwi5vay/rMR8XBErIqIuyJi75p1GyNiZfWxpKZ8fETcHxGPRsRNEbH9FpzDiDNr2lguO20yY8c0E8DYMc1cdtpkx09JkiRJ6jaQMYi3UZm1FCqB8gBg4QD2awKuBI4D2oHlEbEkMx+u2WwF0JKZL0fEp4DLgQ9W13Vm5tReDv0PwBcz88aI+CrwceArm6uPnFBDjWvxig5nE5UkSRoCAxmD+Pma168Bv8rM9gHsNx14LDMfB4iIG4FTgO6AmJl312x/H/Dh/g4YEQEcA/xZteha4CIMiAOzaiHcdQm81A47jYNj58GUM8qulbRVvIWLJEnS0BlIF9OngPsz84eZ+RPg+YjYZwD7jQWerllur5b15ePA7TXLoyOiLSLui4hZ1bJdgfWZ+doAj6kuqxbCbefDS08DWXm+7fxKuTSM9XcLF0mSJA3OQALit4HXa5Y3Vss2J3opy17KiIgPAy3Agprit2dmC5XWwi9FxDsGecxzqgGzbd26dQOoboO76xLY0GMW0w2dlfJ6WLUQvjgJLhpTeTaYaoh4CxdJkqShM5CAuF1m/r5rofp6IBPDtAN71SyPA9b23Cgi3gv8LTAzM1+teZ+11efHgR8A04DngDER0dU1ttdjVve7KjNbMrNlt912G0B1G9xLffQK7qt8KNl6qQL1dasWb+EiSZI0eAMJiOsiYmbXQkScQiWobc5yYEJ11tHtgTOBJbUbRMQ04GtUwuGzNeU7R8QO1ddvAQ4DHs7MBO4GTq9uehZw6wDqop3GDa58KJXdeqmG1jpjIs2jmjYp8xYuxVm8ooPD5i9j/NzvcNj8ZSxe0VF2lSRJ0hAaSEA8F/ifEfFURDwFXAB8cnM7VccJfhpYCjwCLMzMhyLikprAuQDYEfh2j9tZvBNoi4ifUQmE82tmP70A+GxEPEZlTOLVAzrTke7YeTCqR4vKqOZKedHKbL1Uw/MWLvXTNSFQx/pOkj9MCGRIlCSpcUSlUW4AG0bsWN3+N8VWaei1tLRkW1tb2dUoX1mzmH5xUrV7aQ877QV/9fNC39rbH0hD57D5y+joZWzn2DHN/GTuMSXUSJIkDVREPFid46VfA7kP4v8GLs/M9dXlnYH/NzM/t/XVVF1NOaOc21ocO68y5rC2m2kdWi+9/YE0tJwQSJKkxjeQLqYndoVDgMx8EXhfcVVSw5lyBpx8RaXFkKg8n3xF4WHV2x9IQ8sJgSRJanybbUEEmiJih64ZRiOiGdih2Gqp4ZTQemlrhzS0WmdM3KRVHpwQSJKkRjOQgPivwF0R8c3q8tnAtcVVSRoae45p7nW8lK0d0pbp6prtuF5JkhrXZgNiZl4eEauA91K5Uf0dwN5FV0zaWrZ2SENv1rSxBkJJkhrYQFoQAX4NvA6cATwB/N/CaiQNEVs7JEmSpMHpMyBGxH+ncnP72cDzwE1UbnNxdJ3qJm01WzskSZKkgeuvBfE/gB8DJ2fmYwAR8Vd1qZUkSZIkqe76u83F+6l0Lb07Ir4eEcdSGYMoSZIkSWpAfQbEzLwlMz8I7A/8APgrYPeI+EpEHF+n+kmSJEmS6qS/FkQAMvN3mXldZp4EjANWAnMLr5kkSZIkqa42GxBrZeYLmfm1zDymqApJkiRJksoxqIAoSZIkSWpcBkRJkiRJEmBAlCRJkiRVGRAlSZIkSYABUZIkSZJUZUCUJEmSJAEGREmSJElSlQFRkiRJkgQYECVJkiRJVQZESZIkSRJgQJQkSZIkVRkQJUmSJElAwQExIk6IiDUR8VhEzO1l/Wcj4uGIWBURd0XE3tXyqRFxb0Q8VF33wZp9romIJyJiZfUxtchzkCRJkqSRorCAGBFNwJXAicABwOyIOKDHZiuAlsycAtwMXF4tfxn4aGa+CzgB+FJEjKnZrzUzp1YfK4s6B0mSJEkaSYpsQZwOPJaZj2fm74EbgVNqN8jMuzPz5erifcC4avkvMvPR6uu1wLPAbgXWVZIkSZJGvCID4ljg6Zrl9mpZXz4O3N6zMCKmA9sDv6wpvrTa9fSLEbHDUFRWkiRJkka6IgNi9FKWvW4Y8WGgBVjQo3wP4F+AszPz9WrxhcD+wEHALsAFfRzznIhoi4i2devWbdkZSJIkSdIIUmRAbAf2qlkeB6ztuVFEvBf4W2BmZr5aU/7HwHeAz2XmfV3lmflMVrwKfJNKV9Y3yMyrMrMlM1t2283eqZIkSZK0OUUGxOXAhIgYHxHbA2cCS2o3iIhpwNeohMNna8q3B24BvpWZ3+6xzx7V5wBmAT8v8BwkSZIkacTYrqgDZ+ZrEfFpYCnQBHwjMx+KiEuAtsxcQqVL6Y7Atyt5j6cycyZwBnAEsGtEzKkeck51xtLrImI3Kl1YVwLnFnUOkiRJkjSSRGavwwIbSktLS7a1tZVdDUmSJEkqRUQ8mJktm9uuyC6mkiRJkqRhxIAoSZIkSQIMiJIkSZKkKgOiJEmSJAkwIEqSJEmSqgyIkiRJkiTAgChJkiRJqjIgSpIkSZIAA6IkSZIkqcqAKEmSJEkCDIiSJEmSpCoDoiRJkiQJMCBKkiRJkqoMiJIkSZIkwIAoSZIkSaoyIEqSJEmSAAOiJEmSJKnKgChJkiRJAgyIkiRJkqQqA6IkSZIkCTAgSpIkSZKqDIiSJEmSJMCAKEmSJEmqMiBKkiRJkoCCA2JEnBARayLisYiY28v6z0bEwxGxKiLuioi9a9adFRGPVh9n1ZT/j4hYXT3mFRERRZ6DJEmSJI0UhQXEiGgCrgROBA4AZkfEAT02WwG0ZOYU4Gbg8uq+uwB/DxwMTAf+PiJ2ru7zFeAcYEL1cUJR5yBJkiRJI0mRLYjTgccy8/HM/D1wI3BK7QaZeXdmvlxdvA8YV309A/heZr6QmS8C3wNOiIg9gD/OzHszM4FvAbMKPAdJkiRJGjGKDIhjgadrlturZX35OHD7ZvYdW3090GNKkiRJkgZouwKP3dvYwOx1w4gPAy3AkZvZdzDHPIdKV1Te/va3b66ukiRJkjTiFdmC2A7sVbM8Dljbc6OIeC/wt8DMzHx1M/u284duqH0eEyAzr8rMlsxs2W233bb4JCRJkiRppCgyIC4HJkTE+IjYHjgTWFK7QURMA75GJRw+W7NqKXB8ROxcnZzmeGBpZj4D/CYiDqnOXvpR4NYCz0GSJEmSRozCuphm5msR8WkqYa8J+EZmPhQRlwBtmbkEWADsCHy7ereKpzJzZma+EBH/H5WQCXBJZr5Qff0p4BqgmcqYxduRJEmSJG21qEwG2thaWlqyra2t7GpIkiRJUiki4sHMbNncdkV2MZUkSZIkDSMGREmSJEkSYECUJEmSJFUZECVJkiRJgAFRkiRJklRlQJQkSZIkAQZESZIkSVKVAVGSJEmSBBgQJUmSJElVBkRJkiRJEmBAlCRJkiRVbVd2BUaixSs6WLB0DWvXd7LnmGZaZ0xk1rSxZVdLkiRJ0ghnQKyzxSs6uHDRajo3bASgY30nFy5aDWBIlCRJklQqu5jW2YKla7rDYZfODRtZsHRNSTWSJEmSpAoDYp2tXd85qHJJkiRJqhcDYp3tOaZ5UOWSJEmSVC8GxDprnTGR5lFNm5Q1j2qidcbEkmokSZIkSRVOUlNnXRPROIupJEmSpG2NAbEEs6aNNRBKkiRVbdiwgfb2dl555ZWyqyINe6NHj2bcuHGMGjVqi/Y3IEqSJKlU7e3tvPnNb2afffYhIsqujjRsZSbPP/887e3tjB8/fouO4RhESZIkleqVV15h1113NRxKWyki2HXXXbeqNd6AKEmSpNIZDqWhsbX/lgyIkiRJGvE+9rGP8da3vpVJkyb1uU1mcv7557PffvsxZcoUfvrTn3avu/baa5kwYQITJkzg2muv3ez7/eAHP+Ckk04akrpviWuuuYa1a9eW9v5D7Y477mDixInst99+zJ8/v9dtvvCFL3DAAQcwZcoUjj32WH71q191r2tqamLq1KlMnTqVmTNnbvb9nnzyyX6vlaItXryYhx9+uJBjGxAlSZI0rCxe0cFh85cxfu53OGz+Mhav6NjqY86ZM4c77rij321uv/12Hn30UR599FGuuuoqPvWpTwHwwgsvcPHFF3P//ffzwAMPcPHFF/Piiy9uUT1ee+21LdpvsEoNiKsWwhcnwUVjKs+rFm7V4TZu3Mh5553H7bffzsMPP8wNN9zQa3iaNm0abW1trFq1itNPP52/+Zu/6V7X3NzMypUrWblyJUuWLNniutTr92dAlCRJkqiEwwsXraZjfScJdKzv5MJFq7c6JB5xxBHssssu/W5z66238tGPfpSI4JBDDmH9+vU888wzLF26lOOOO45ddtmFnXfemeOOO67XsHnHHXew//77c/jhh7No0aLu8osuuohzzjmH448/no9+9KO88sornH322UyePJlp06Zx9913A5VQd8opp3DCCScwceJELr744u5jfOELX2DSpElMmjSJL33pS8AbW7k+//nPc9FFF3HzzTfT1tbGhz70IaZOnUpnZ+dW/ewGZdVCuO18eOlpICvPt52/VSHxgQceYL/99mPfffdl++2358wzz+TWW299w3ZHH300f/RHfwTAIYccQnt7+6De58EHH+TAAw/k0EMP5corr+wuv+aaa/jABz7AySefzPHHH09m0trayqRJk5g8eTI33XQTUGk1PuKIIzj11FM54IADOPfcc3n99dcBuOGGG5g8eTKTJk3iggsu6D72jjvu2P365ptvZs6cOfz7v/87S5YsobW1lalTp/LLX/5yUOexOYUGxIg4ISLWRMRjETG3l/VHRMRPI+K1iDi9pvzoiFhZ83glImZV110TEU/UrJta5DlIkiRp27Fg6Ro6N2zcpKxzw0YWLF1T+Ht3dHSw1157dS+PGzeOjo6OPstrvfLKK3ziE5/gtttu48c//jG//vWvN1n/4IMPcuutt3L99dd3h4/Vq1dzww03cNZZZ3VPOvLAAw9w3XXXsXLlSr797W/T1tbGgw8+yDe/+U3uv/9+7rvvPr7+9a+zYsWKPs/j9NNPp6Wlpfs4zc3NW/2zGbC7LoENPQLphs5K+RYayM+/p6uvvpoTTzyxe/mVV16hpaWFQw45hMWLF/e6z9lnn80VV1zBvffe+4Z19957L9deey3Lli1j0aJFrFy5kp/97Gd8//vfp7W1lWeeeQao/P7+8R//kdWrV/PLX/6SRYsWsXbtWi644AKWLVvGypUrWb58eZ91AHjPe97DzJkzWbBgAStXruQd73hHv+c6WIUFxIhoAq4ETgQOAGZHxAE9NnsKmANcX1uYmXdn5tTMnAocA7wM3FmzSWvX+sxcWdQ5SJIkaduydn3vrV19lQ+lzHxDWUT0WV7rP/7jPxg/fjwTJkwgIvjwhz+8yfqZM2d2B7V77rmHj3zkIwDsv//+7L333vziF78A4LjjjmPXXXelubmZ0047jXvuuYd77rmHU089lTe96U3suOOOnHbaafz4xz8eknMeci/10WrXV/kADOTnX+tf//VfaWtro7W1tbvsqaeeoq2tjeuvv56//Mu/fEOr3EsvvcT69es58sgjAbp/P126WpCh8vubPXs2TU1N7L777hx55JEsX74cgOnTp7PvvvvS1NTE7Nmzueeee1i+fDlHHXUUu+22G9tttx0f+tCH+NGPfrRlP4whUGQL4nTgscx8PDN/D9wInFK7QWY+mZmrgNf7Oc7pwO2Z+XJxVa2zIe53LUmSNFLsOab31q6+yofSuHHjePrpp7uX29vb2XPPPfss76m/0PKmN72p+3VvgaevY/QVUAG222677i6MwFbd+mDI7DRucOUDMNCfP8D3v/99Lr30UpYsWcIOO+zQXd61/b777stRRx31hhbYzKz776/n9vX6/RUZEMcCT9cst1fLButM4IYeZZdGxKqI+GJE7NDbTtusAvpdS5IkjRStMybSPKppk7LmUU20zphY+HvPnDmTb33rW2Qm9913HzvttBN77LEHM2bM4M477+TFF1/kxRdf5M4772TGjBmb7Lv//vvzxBNPdLdM3XBDz6+3f3DEEUdw3XXXAfCLX/yCp556iokTK+f3ve99jxdeeIHOzk4WL17MYYcdxhFHHMHixYt5+eWX+d3vfsctt9zCn/zJn7D77rvz7LPP8vzzz/Pqq6/yb//2b93v8eY3v5nf/OY3Q/0j2rxj58GoHmF+VHOlfAsddNBBPProozzxxBP8/ve/58Ybb+x1JtIVK1bwyU9+kiVLlvDWt761u/zFF1/k1VdfBeC5557jJz/5CQccsGnHxzFjxrDTTjtxzz33AHT/fnpzxBFHcNNNN7Fx40bWrVvHj370I6ZPnw5Uupg+8cQTvP7669x0000cfvjhHHzwwfzwhz/kueeeY+PGjdxwww3dLZW77747jzzyCK+//jq33HJL93sU+fsrMiD2FrH7jse9HSBiD2AysLSm+EJgf+AgYBfggl52JSLOiYi2iGhbt27dYN62WAX0u5YkSRopZk0by2WnTWbsmGYCGDummctOm8ysaVvSDvEHs2fP5tBDD2XNmjWMGzeOq6++GoCvfvWrfPWrXwXgfe97H/vuuy/77bcfn/jEJ/inf/onAHbZZRf+7u/+joMOOoiDDjqIefPmvWHCm9GjR3PVVVfxp3/6pxx++OHsvffefdblL/7iL9i4cSOTJ0/mgx/8INdcc013a9fhhx/ORz7yEaZOncr73/9+WlpaePe7382cOXOYPn06Bx98MH/+53/OtGnTGDVqFPPmzePggw/mpJNOYv/99+9+jzlz5nDuuefWf5KaKWfAyVfATnsBUXk++YpK+Rbabrvt+PKXv8yMGTN45zvfyRlnnMG73vUuAObNm9c9K2lrayu//e1v+cAHPrDJ7SweeeQRWlpaOPDAAzn66KOZO3fuGwIiwDe/+U3OO+88Dj300H7HbZ566qlMmTKFAw88kGOOOYbLL7+ct73tbQAceuihzJ07l0mTJjF+/HhOPfVU9thjDy677DKOPvpoDjzwQN797ndzyimVjpfz58/npJNO4phjjmGPPfbofo8zzzyTBQsWMG3atCGfpCb6a9LcqgNHHApclJkzqssXAmTmZb1sew3wb5l5c4/yzwDvysxz+niPo4C/zsx+byLT0tKSbW1tW3IaQ++iMfSekwMuWl/v2kiSJJXukUce4Z3vfGfZ1djmXXPNNbS1tfHlL3+57KpoC/zgBz/g85///CYtuUXp7d9URDyYmS2b27fIFsTlwISIGB8R21PpKjrYm4rMpkf30mqrIlHpkDsL+PkQ1LV+Cuh3LUmSJElDobCAmJmvAZ+m0j30EWBhZj4UEZdExEyAiDgoItqBDwBfi4iHuvaPiH2AvYAf9jj0dRGxGlgNvAX4X0WdQyEK6HctSZKkxjdnzhxbD4exo446qi6th1truyIPnpnfBb7bo2xezevlQK9NZ5n5JL1MapOZxwxtLeusq3/1XZdUpvPdaVwlHG5Fv2tJkiRJGgqFBkT1YcoZBkJJkqQam7uNgKSB2do5ZoocgyhJkiRt1ujRo3n++ee3+outNNJlJs8//zyjR4/e4mPYgihJkqRSjRs3jvb2drapW5NJw9To0aMZN27LJ8A0IEqSJKlUo0aNYvz48WVXQxJ2MZUkSZIkVRkQJUmSJEmAAVGSJEmSVBUjYbaoiFgH/KrsevTiLcBzZVdCDcvrS0Xy+lKRvL5UNK8xFWlbvb72zszdNrfRiAiI26qIaMvMlrLrocbk9aUieX2pSF5fKprXmIo03K8vu5hKkiRJkgADoiRJkiSpyoBYrqvKroAamteXiuT1pSJ5faloXmMq0rC+vhyDKEmSJEkCbEGUJEmSJFUZEEsQESdEtXuVpwAABJhJREFUxJqIeCwi5pZdHzWeiHgyIlZHxMqIaCu7PhreIuIbEfFsRPy8pmyXiPheRDxafd65zDpq+Orj+rooIjqqn2ErI+J9ZdZRw1dE7BURd0fEIxHxUER8plruZ5i2Wj/X17D+DLOLaZ1FRBPwC+A4oB1YDszOzIdLrZgaSkQ8CbRk5rZ4Dx4NMxFxBPBb4FuZOaladjnwQmbOr/6ha+fMvKDMemp46uP6ugj4bWZ+vsy6afiLiD2APTLzpxHxZuBBYBYwBz/DtJX6ub7OYBh/htmCWH/Tgccy8/HM/D1wI3BKyXWSpD5l5o+AF3oUnwJcW319LZX/EKVB6+P6koZEZj6TmT+tvv4N8AgwFj/DNAT6ub6GNQNi/Y0Fnq5ZbqcBLiRtcxK4MyIejIhzyq6MGtLumfkMVP6DBN5acn3UeD4dEauqXVDt/qetFhH7ANOA+/EzTEOsx/UFw/gzzIBYf9FLmf18NdQOy8x3AycC51W7cEnScPEV4B3AVOAZ4B/LrY6Gu4jYEfi/wF9m5n+VXR81ll6ur2H9GWZArL92YK+a5XHA2pLqogaVmWurz88Ct1Dp2iwNpf+sjr3oGoPxbMn1UQPJzP/MzI2Z+TrwdfwM01aIiFFUvrxfl5mLqsV+hmlI9HZ9DffPMANi/S0HJkTE+IjYHjgTWFJyndRAIuJN1YHSRMSbgOOBn/e/lzRoS4Czqq/PAm4tsS5qMF1f3KtOxc8wbaGICOBq4JHM/ELNKj/DtNX6ur6G+2eYs5iWoDrV7ZeAJuAbmXlpyVVSA4mIfam0GgJsB1zvNaatERE3AEcBbwH+E/h7YDGwEHg78BTwgcx0ohENWh/X11FUumYl8CTwya7xYtJgRMThwI+B1cDr1eL/SWWcmJ9h2ir9XF+zGcafYQZESZIkSRJgF1NJkiRJUpUBUZIkSZIEGBAlSZIkSVUGREmSJEkSYECUJEmSJFUZECVJGqCI2BgRK2sec4fw2PtExLC6V5YkqfFsV3YFJEkaRjozc2rZlZAkqSi2IEqStJUi4smI+IeIeKD62K9avndE3BURq6rPb6+W7x4Rt0TEz6qP91QP1RQRX4+IhyLizohoLu2kJEkjkgFRkqSBa+7RxfSDNev+KzOnA18GvlQt+zLwrcycAlwHXFEtvwL4YWYeCLwbeKhaPgG4MjPfBawH3l/w+UiStInIzLLrIEnSsBARv83MHXspfxI4JjMfj4hRwK8zc9eIeA7YIzM3VMufycy3RMQ6YFxmvlpzjH2A72XmhOryBcCozPxfxZ+ZJEkVtiBKkjQ0so/XfW3Tm1drXm/EuQIkSXVmQJQkaWh8sOb53urrfwfOrL7+EHBP9fVdwKcAIqIpIv64XpWUJKk//mVSkqSBa46IlTXLd2Rm160udoiI+6n88XV2tex84BsR0QqsA86uln8GuCoiPk6lpfBTwDOF116SpM1wDKIkSVupOgaxJTOfK7sukiRtDbuYSpIkSZIAWxAlSZIkSVW2IEqSJEmSAAOiJEmSJKnKgChJkiRJAgyIkiRJkqQqA6IkSZIkCTAgSpIkSZKq/n9C/djIySigVwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1080x1080 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot train and validation accuracies of the two models\n",
    "\n",
    "train_accs = []\n",
    "val_accs = []\n",
    "for dropout in dropout_choices:\n",
    "    solver = solvers[dropout]\n",
    "    train_accs.append(solver.train_acc_history[-1])\n",
    "    val_accs.append(solver.val_acc_history[-1])\n",
    "\n",
    "plt.subplot(3, 1, 1)\n",
    "for dropout in dropout_choices:\n",
    "    plt.plot(solvers[dropout].train_acc_history, 'o', label='%.2f dropout' % dropout)\n",
    "plt.title('Train accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend(ncol=2, loc='lower right')\n",
    "  \n",
    "plt.subplot(3, 1, 2)\n",
    "for dropout in dropout_choices:\n",
    "    plt.plot(solvers[dropout].val_acc_history, 'o', label='%.2f dropout' % dropout)\n",
    "plt.title('Val accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend(ncol=2, loc='lower right')\n",
    "\n",
    "plt.gcf().set_size_inches(15, 15)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inline Question 2:\n",
    "Compare the validation and training accuracies with and without dropout -- what do your results suggest about dropout as a regularizer?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Answer:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inline Question 3:\n",
    "Suppose we are training a deep fully-connected network for image classification, with dropout after hidden layers (parameterized by keep probability p). How should we modify p, if at all, if we decide to decrease the size of the hidden layers (that is, the number of nodes in each layer)?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Answer:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
